feat: add tautobase#658
Conversation
ElliottKasoar
added
the
new benchmark
|
Hey @lwehrhan thanks for the pr! can you attach the data for us to upload please?
docs: https://ddmms.github.io/ml-peg/developer_guide/filter.html |
|
I've added similar changes to the folmsbee benchmark, and apart from the comments/conflicts i think we're basically good to go. @ElliottKasoar could you take a look? |
ElliottKasoar
left a comment
ElliottKasoar
left a comment
There was a problem hiding this comment.
This looks great! Just a couple of minor suggestions/questions, plus if you could rebase/resolve conflicts with due to merging #429?
| from ase import Atoms | ||
| from ase.calculators.calculator import Calculator | ||
| from ase.io import write | ||
| from mlipaudit.benchmarks.tautomers.tautomers import TautomersModelOutput |
There was a problem hiding this comment.
Add pytest.importorskip?
| from typing import Any | ||
| from warnings import warn | ||
|
|
||
| from mlipaudit.benchmarks.tautomers.tautomers import TautomersModelOutput |
There was a problem hiding this comment.
pytest.importorskip
| List of all tautomer pair structure IDs. | ||
| """ | ||
| for model_name in MODELS: | ||
| path = CALC_PATH / model_name / "model_output.json" |
There was a problem hiding this comment.
See changes to Folmsbee - I'd recommend using "mock" as the source of truth, and raising an error telling people to run it if it's missing
|
|
||
|
|
||
| @pytest.fixture | ||
| def struct_info() -> dict: |
There was a problem hiding this comment.
See changes to Folsmbee - if we save the json file downloaded by the calc to the calc outputs, we can just use that, and make it more self contained
|
|
||
| structs_dir = OUT_PATH / "mock" | ||
| structs_dir.mkdir(parents=True, exist_ok=True) | ||
| for structure_id, pair in benchmark._tautomers_data.items(): |
There was a problem hiding this comment.
Just to check, the way you construct the Atoms here doesn't work quite the same way as in Folmsbee? There's no way to have a shared function for these? It's fine if not
| 1. Reaction energy MAE | ||
|
|
||
| For each pair the reaction energy is the energy difference between the two | ||
| tautomers. The mean absolute error (MAE) between the predicted and reference | ||
| reaction energies is reported in kcal/mol, averaged across all pairs. Pairs on | ||
| which inference fails are excluded from the average. |
There was a problem hiding this comment.
Can you add the second metric here?
4 participants
Pre-review checklist for PR author
PR author must check the checkboxes below when creating the PR.
Summary
The tautobase is a dataset of 1,391 tautomer pairs of organic small molecules. The geometries were refined at xtb level of theory. QM single point energies (reference for this benchmark) were re-computed by InstaDeep in-house at ωB97M-D3(BJ)/def2-TZVPPD level of theory.
Linked issue
Resolves #641
Progress
Testing
New decorators/callbacks